(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(11) 



EP1 530 158 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

11.05.2005 Bulletin 2005/19 

(21) Application number: 04105511 .2 

(22) Date of filing: 04.1 1 .2004 


(51) IntCI ?: G06T 7/00 


(84) Designated Contracting States: 


(72) 


Inventors: 


AT BE BG CH CY CZ DE DK EE ES Fl FR GB GR 


• 


Kakiuchi, Takashi 


HU IE IS IT LI LU MC NL PL PT RO SE SI SK TR 




600-8530, Kyoto (JP) 


Designated Extension States: 


• 


Senga, Masahiro 


AL HR LTLV MK YU 




600-8530, Kyoto (JP) 




• 


Hosoi, Satoshi 


(30) Priority 05.11.2003 JP 2003375804 




600-8530, Kyoto (JP) 


16.09.2004 JP 2004270155 


• 


Matsuoka, Miki 






600-8530, Kyoto (JP) 


(71) Applicant. Omron Corporation 






Kyoto-shi, Kyoto 600-8530 (JP) 


(74) 


Representative: Weihs, Bruno 


Designated Contracting States: 




Osha & May 


AT BE DE 




121, avenue des Champs Elysees 






75008 Paris (FR) 



(54) Pupil color estimating device 



(57) The present invention aims to estimate from an 
image an original pupil color of a subject person in an 
image in which an abnormality such as a red-eye or 
gold-eye occurs. Further, the present invention aims to 
judge whether an image correction is required or not and 
to produce an image having a natural pupil color in the 
image correction by estimating the original pupil color 
from the image. The present invention provides a pupil 
color estimating device wherein a skin color of a subject 
person is specified from an inputted image, regardless 
of whether an abnormality such as a red-eye or gold- 
eye occurs, whereupon this skin color is substituted in 
an approximate equation based upon a statistical result 
carried out in advance, thereby estimating a normal 
colorcomponent of a pupil from the skin color of the sub- 
ject person. 
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Description 

BACKGROUND OF THE INVENTION 

5 Technical Field 

[0001 ] The present invention relates to a technique that is effective when it is applied to an imaging device for image- 
capturing an image (particularly, an image in which a person's face is image-captured), an information processing 
device or an output device that handles an image, a software or the like. 

w 

Related Prior Art 

[0002] Conventionally, a red-eye phenomenon wherein a pupil or whole pupil of the image-captured person is image- 
captured to be red becomes a problem, regardless of a digital camera or silver-halide camera. As a similar problem, 

15 there is gold-eye phenomenon in which a pupil or whole pupil of a person is image-captured to be gold. The red-eye 
or gold-eye phenomenon described above (hereinafter, the explanation is made by taking the red-eye phenomenon 
as an example) is a phenomenon caused such that flash light incident on a bulb of an eye from a pupil region is reflected 
on the retina. Since the distance between the optical axis of a camera and a flash becomes close to each other recently 
due to the miniaturization of a camera, the rate of occurrence of the red-eye phenomenon increases, which becomes 

20 a problem in particular. 

[0003] A red-eye preventing flash has been proposed as a technique for preventing and reducing the above-men- 
tioned red-eye phenomenon. This is a technique wherein a flash is lightly emittedimmediatelybefore the image-capture, 
and then, the flash is again emitted. According to this technique, the pupil of the image-captured person is contracted 
by the first emission , so that the occurrence of the red-eye phenomenon by the flash after that can be prevented and 

25 reduced 

[0004] However, when the red-eye phenomenon occurs, the red eye is required to be image-corrected, so long as 
the image cannot be retaken. As an image-correction technique as described above, there has conventionally been 
proposed a method wherein a pixel having the red-eye phenomenon is designated and the color of this pixel is changed 

to correct the image. 

30 [0005] Further, in the case where a color of a pixel having the red-eye phenomenon occurring is changed, it is required 
that the pupil color after the change is determined. As a technique for determining the pupil color, there has conven- 
tionally been proposed a technique wherein a user selects a pupil color recorded in advance every race for changing 
the color of the pixel of the pupil by using this selected color. Moreover, as a similar technique, there has been proposed 
a method wherein a natural-looking pupil color is selected among a plurality of patterns such as "a pattern turned to 

35 look like an European" or "a pattern of Orientals", to thereby make a change by using this color. 

[0006] However, the image of the pupil having the red-eye phenomenon already occurring loses the original color 
of the pupil. Therefore, it is difficult to estimate the original color of the pupil from the image of the pupil having the red- 
eye phenomenon. If the original color of the pupil cannot be estimated, it is naturally difficult to produce a natural image 
even if the correction is intended to be executed by using a conventional technique. 

40 [0007] Further even if the above-mentioned techniques are applied, a user is required to estimate the race of the 
subject by a visual observation. In general . there are not so many users having knowledge about the race, so that 
estimating the pupil color based upon the race is a difficult and troublesome process for the user. 
[0008] Additionally, it is considered that, as a method for correcting the color of the pupil, a reddish section is simply 
removed from the color of the pupil having the red-eye phenomenon, not replacing the color of the pupil by estimating 

^5 the original color of the pupil. However, there is a problem upon performing such a correcting method. A person's sight 
has a feature of judging colorfulness by a color contrast. Therefore, in the case where a subject is a ruddy-cheeked 
person or a subject is photographed under a red illumination, there is a possibility that an unnatural image is produced 
on the contrary only by performing the correction for simply removing the reddish section from the color of the pupil. 

50 DISCLOSURE OF THE INVENTION 

[0009] An object of the present invention is to provide a technique capable of estimating a color of a pupil of an 
image-captured person (an original color of a pupil in the captured image) from an image of the image-captured person, 
in the case where it is recognized that an abnormality which can appear in an image of a pupil region such as a red- 
55 eye phenomenon, does not occur. 

[0010] In the following explanation, a skin color means a skin color of anyone and any race, and it is not limited to 
a skin color of a specific person or specific race. 
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[First Aspect] 

[0011] In order to solve the above-mentioned problem, the present invention has a configuration described later. The 
first aspect is a pupil color estimating device and includes skin color specifying means and pupil color estimating means. 

5 [001 2] The skin color specifying means specifies a skin color of a subject person in an inputted image. The skin color 
specifying means may be configured to specify a skin color manually by a user. The skin color specifying means may 
be configured such that a color designatedby a user from the inputted image, a pixel color in the image designated by 
the user (i.e., a measured value at the section designated by the user) or the like is specified as the skin color of the 
person Further the skin color specifying means may be configured to specify a skin color based upon the input from 

10 the user. The skin color specifying means may be configured to specify the statistical result (average, mode, interme- 
diate value, or the like) of a pixel color at the region in the image designated by the user as the skin color of the person. 
[0013] Further the skin color specifying means may be configured to specify a skin color independently of the input 
from the user. The skin color specifying means having such configuration will be explained in the second aspect. 
[0014] The pupil color estimating means estimates a color of the pupil of the person that should be shown in the 

75 image based upon the skin color specified by the skin color specifying means. The "color of the pupil of the person 
that should be shown in the image" estimated by the pupil color estimating means is not the color itself of the pupil 
region in the inputted image, but indicates a color of the pupil recognized to be shown in the image that is image- 
captured without having an abnormality such as a red-eye or gold-eye phenomenon occurring, in other words, indicates 
a color of the pupil that should originally be outputted. 

20 [0015] In the first aspect having the above-mentioned configuration, the skin color of a subject person is specified 
from the inputted image, and based upon this skin color, the color of the pupil of the person is estimated. Therefore, it 
is possible to estimate the color of the pupil that is natural and has no sense of incongruity with the color contrast 
considered for the skin color in this image. Accordingly, the original color of the pupil (the color of the pupil recognized 
to be shown (outputted) in the case where an abnormality does not occur) can be estimated from the image in which 

25 the original color of the pupil is lost due to the occurrence of the red-eye phenomenon or gold-eye phenomenon. 

[Second Aspect) 

[0016] The second aspect is a pupil color estimating device and is configured to be provided with face detecting 
so means, skin color specifying means and the pupil color estimating means in the first aspect. 

[0017] The face detecting means detects a face region of a subj ect person in an inputted image. "The face region" 
means a region including a part of the face or whole face of the subject person. 

[0018] The skin color specifying means in the second aspect specifies the skin color of the person from a pixel color 
included in the face region detected by the face detecting means. The skin color specifying means may be configured 
35 to specify the statistical result (average, mode, intermediate value, or the like) of a pixel color at the face region detected 
by the face detecting means as the skin color of the person. 

[001 9] In the second aspect having the above-mentioned configuration, the skin color of a subject person is specified 
from the inputted image, and based upon this skin color, the color of the pupil of the person is estimated. Therefore, 
like the first aspect it is possible to estimate the color of the pupil that is natural and has no sense of incongruity with 
-to the color contrast considered for the skin color in this image. Accordingly, the original color of the pupil (the color of 
the pupil recognized to be shown (outputted) in the case where an abnormality does not occur) can be estimated from 
the image in which the original color of the pupil is lost due to the occurrence of the red-eye phenomenon or gold-eye 
phenomenon. 

[0020] In the second aspect, the skin color specifying means performs a processing based upon the detected result 
J5 by the face detecting means, thereby being capable of specifying the skin color independently of the input from the 
user (i.e., even though a color or a region in the image for specifying the color is not designated by the user upon the 
specifying process of the skin color). Therefore, a troublesome job of the user can be saved. 

[Third Aspect] 

50 

[0021] The third aspect is further provided with pupil region specifying means and judging means, in addition to the 
components of the first or second aspect. The pupil region specifying means specifies the region of the pupil of a 
subject person in an image. The pupil region specifying means may be configured to specify the region of the pupil 
based upon the input from a user, or may be configured to specify the region of the pupil independently of the input 

55 from the user. 

[0022] The judgingmeans compares the color of the pupil estimated by the pupil color estimating means and the 
color of the pupil region specified by the pupil region specifying means. The judging means judges whether an abnor- 
mality occurs or not at the pupil section in the image based upon this comparison result. The abnormality occurring at 
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the pupil section in the image means, for example, a red-eye phenomenon or gold-eye phenomenon, which is an 
abnormality occurring on a part of the pupil section or whole color. 

[0023] In the third aspect having the above-mentioned configuration, the color of the pupil estimated from the skin 
color and the color of the pupil region specified from the image are compared to judge whether an abnormality occurs 
5 or not at the pupil section in the image. Accordingly, the user can easily determine a measure to this image based upon 
this judging result. For example, in the case where the user thinks of performing a correction to the image on which 
an abnormality occurs, he/she can judge whether the correction is made or not based upon this judging result. Further, 
in the case where the user thinks of retaking an image when an abnormality occurs on the image-captured image, for 
example, he/she can judge whether the image is recaptured or not based upon this judging result. 

10 

[Fourth Aspect] 

[0024] The fourth aspect is further provided with pupil detecting means and judging means in addition to the com- 
ponents in the second aspect. 

*5 [0025] The pupil detecting means detects the pupil region of the person detected by the face detecting means. "The 
pupil region" means a region including a part of the pupil or whole pupil of the subject person. 

[0026] The judging means compares the color of the pupil estimated by the pupil color estimating means and the 
color specified from the pupil region detected by the pupil detecting means, thereby judging whether an abnormality 
occurs or not on the color of the pupil of the person in the image that is to be processed. For example, the judging 
20 means makes a judgement by comparing the color of each pixel in the detected pupil region and the estimated pupil 
color. Moreover, the judging means may specify one representative value from the color of each pixel in the detected 
pupil region by a statistical process, thereby making a judgement by comparing this representative value and the 
estimated pupil color. 

[0027] In the fourth aspect having the above-mentioned configuration, the color of the pupil estimated from the skin 
25 color and the color in the detected pupil region are compared to judge whether an abnormality occurs or not at the 
pupil section in the image. Accordingly, like the third aspect, the user can easily determine a measure to this image 
based upon this judging result. Further, in the fourth aspect, the judging means performs a processing based upon the 
detected result by the pupil detecting means, thereby being capable of specifying the color of the pupil independently 
of the input from the user (i.e., even though a color or a region in the image for specifying the color is not designated 
30 by the user upon the specifying process of the pupil color). Therefore, a troublesome job of the user can be saved. 

[Fifth Aspect] 

[0028] The fifth aspect is further provided with correcting means in addition to the components in the first or second 
55 aspect. The correcting means corrects the color of the pupil of the person based upon the color of the pupil estimatedby 
the pupil estimating means. The correcting means may be configured to make a correction based upon an input from 
a user or may be configured to make a correction independently of the input from the user. For example, the correcting 
means may be configured to make a correction to a pixel of the pupil designated by the user. Further, the correcting 
means may be configured, for example, to make a correction to the pupil region designated by the user. Moreover, the 
40 correcting means may be configured, for example, to make a correction to the pupil region determined based upon 
the color or pixel designated by the user. 

[0029] In the fifth aspect, the color of the pupil of the person in the image is corrected based upon the pupil color 
estimated by the pupil color estimating means. Therefore, a correction considering the color contrast, i.e., a natural 
correction having no sense of incongruity can be made to the pupil. Specifically, correction quality can be enhanced. 

45 

[Sixth Aspect] 

[0030] The sixth aspect is further provided with the pupil detecting means in the fourth aspect and correcting means, 
in addition to the components in the second aspect. 

50 [0031 ] The correcting means in the sixth aspect corrects the color of the pupil with respect to the pupil region detected 
by the pupil detecting means based upon the pupil color estimated by the pupil color estimating means. 
[0032] In the sixth aspect, the pupil color of the person in the image is corrected based upon the pupil color estimated 
by the pupil color estimatingmeans, like the fifth aspect. Therefore, a correction considering the color contrast, i.e., a 
natural correction having no sense of incongruity can be made to the pupil. Specifically, correction quality can be 

55 enhanced. 

[0033] Further in the sixth aspect, the correctingmeans performs a proces sing based upon the detected result by 
the pupil detecting means, thereby being capable of correcting the color of the pupil independently of the input from 
the user. Therefore, it is unnecessary for the user to designate a pixel or a region to the pupil color estimating device, 
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which makes it possible to save a troublesome job of the user. 
[Seventh Aspect] 

5 [0034] The seventh aspect is further provided with correcting means in addition to the components in the third or 
fourth aspect. In the seventh aspect, the correcting means corrects the pupil color of the person based upon the pupil 
color estimated by the pupil color estimating means, in the case where an abnormality is judged to occur on the pupil 
color by the judging means. Specifically, the correcting means in the seventh aspect operates as the correcting means 
provided at the fifth aspect, in the case where the judging means judges that an abnormality occurs. 

10 [0035] According to the seventh aspect, the correcting process is executed based upon the judging result by the 
judging means, thereby being capable of saving useless process such as the execution of the correcting process to 
an image to which the correcting process is unnecessary. 

[0036] Further, the correcting means provided at the seventh aspect operates as the correcting means provided at 
the fifth aspect, so that the pupil color of the person in the image is corrected based upon the pupil color estimated by 
*5 the pupil color estimating means. Therefore, a correction considering the color contrast, i.e., a natural correction having 
no sense of incongruity can be made to the pupil. Specifically, correction quality can be enhanced. 

[Eighth Aspect] 

20 [0037] The eighth aspect is further provided with correcting means in addition to the components in the fourth aspect. 
The correcting means in the eighth aspect executes the correcting process, like the correcting means in the seventh 
aspect in the case where an abnormality is judged to occur on the pupil color by the judging means. Further, the 
correcting means in the eighth aspect corrects the pupil color with respect to the pupil region detected by the pupil 
detecting means based upon the pupil color estimated by the pupil color estimating means, like the correcting means 

25 in the sixth aspect. 

[0038] The eighth aspect can save useless process such as the execution of the correcting process to an image to 
which the correcting process is unnecessary, like the seventh aspect. 

[0039] Moreover, the correcting means provided at the eighth aspect operates as the correcting means provided at 
the sixth aspect for executing the correcting process, thereby affording the following effects. Specifically, the pupil color 

30 of the person in the image is corrected based upon the pupil color estimated by the pupil color estimating means, like 
the fifth aspect. Therefore, a correction considering the color contrast, i.e., a natural correction having no sense of 
incongruity can be made to the pupil. Specifically, correction quality can be enhanced. Further, like the sixth aspect, 
the correcting means performs a processing based upon the detected result by the pupil detecting means, thereby 
being capable of correcting the color of the pupil independently of the input from the user. Therefore, it is unnecessary 

35 for the user to designate a pixel or a region to the pupil color estimating device, which makes it possible to save a 
troublesome job of the user. 

[Ninth Aspect] 

40 [0040] The ninth aspect has the judging means in the third, fourth, seventh and eighth aspects having the following 
configuration. In the ninth aspect, the judging means judges whether a red-eye occurs or not as an example of an 
abnormality occurring on the image. More specifically, the judging means compares the red section in the pupil color 
estimatedby the pupil color estimating means and the red section in the color specified from the pupil region detected 
by the oupil detecting means, to thereby judge whether the pupil section in the image becomes a red eye or not. 

45 

[Tenth Aspect] 

[0041] In the tenth aspect, the correcting means in the fifth to eighth aspects is configured as follows. The correcting 
means makes a correction to a pixel, which is a subject to be corrected, with the brightness of this pixel maintained. 
50 In the case where the pupil detecting means is provided like the eighth aspect, the correcting means in the tenth aspect 
makes a correction to a pixel in the pupil region detected by the pupil detecting means with the brightness of this pixel 

maintained. 

[0042] Incidentally, a man's sight generally catches a spherical shape of a bulb or a sense of wettability on the surface 
of the bulb by a so-called catch light (regular reflection phenomenon caused on the surface of a bulb). Therefore, an 
55 image of a pupil having no catch light is sometimes grasped as an unnatural image for a person. 

[0043] Further, a catch light itself is lost from an image of a pupil having caused thereon a red-eye phenomenon or 
gold-eye phenomenon, but the distribution of the brightness in this image shows the same distribution in the case of 
tne catch light in most cases. 
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[0044] A correction is made while keeping the brightness of the pixel as described above in the tenth aspect. There- 
fore, it is possible to naturally express a spherical shape of a bulb or a sense of wettability on the surface of the bulb 
in the image of the pupil after the correction without losing a so-called catch light. Accordingly, high-quality image 
correction can be realized. 

5 

[Eleventh Aspect] 

[0045] The eleventh aspect is configured such that, in the aforesaid each aspect, the pupil color estimating means 
estimates a pupil color based upon a correlation between a skin color and a pupil color, this correlation obtained based 

10 upon the result of a statistical process of a skin color and a pupil color obtained from an image including a subject 
person that is recognized to have no abnormality occurring on the pupil section. "The result of a statistical process" 
here means the result of a statistical process executed with respect to a skin color and a pupil color in an image that 
includes a subject person that is recognized to have no abnormality occurring on the image of the pupil. The pupil color 
estimating means may be configured to estimate a pupil colorbasedupon a table that is made basedupon the aforesaid 

*5 correlation and wherein a skin color and a pupil color are associatedwith each other. Further, the pupil color estimating 
means may be configured to estimate a pupil color based upon a relational expression between a skin color and a 
pupil color obtained based upon the aforesaid correlation. 

[0046] This configuration allows the pupil estimating means to estimate a more natural pupil color supported by the 
result of the statistical process based upon the skin color of the subject person. 

20 

[Twelfth Aspect] 

[0047] The twelfth aspect is further provided with an approximate equation storing unit, a race estimating unit and a 
selecting unit in the above-mentioned each embodiment. The approximate equation storing unit stores the approximate 

25 equation used for estimating a pupil color so as to associate with the race. Such approximate equation is an approximate 
equation obtained by the result of the statistical process shown in the eleventh embodiment and is obtained in advance. 
The race estimating unit estimates the race of the subject person. Any known race estimating technique can be applied 
to the race estimating unit. The selecting unit selects the approximate equation stored in the approximate equation 
storing unit so as to associate with the estimated race. Further, in the twelfth aspect, the pupil color estimating unit 

30 estimates a pupil color based upon the specified skin color and using the selected approximate equation. 

[0048] That a pupil color is different depending upon the race is based upon an experiential fact. Accordingly, esti- 
mating a pupil color by using the same approximate equation possibly degrades the precision, in the case where a 
different race becomes a subject. However, the twelfth aspect makes it possible to estimate a more suitable pupil color 
according to the race of the subject. Therefore, more natural pupil color can be estimated. 

35 

[Thirteenth Aspect] 

[0049] The thirteenth aspect is further provided with an approximate equation storing unit, an environment estimating 
unit and a selecting unit in the above-mentioned each aspect. The approximate equation storing unit stores the ap- 

40 proximate equation used for estimating a pupil color so as to associate with the environment. Such an approximate 
equation is an approximate equation obtained by the result of the statistical process shown in the eleventh aspect and 
is obtained in advance. The environment estimating unit estimates the environment in which this device operates or 
the environment in which the inputted image is captured. The environment estimating unit may estimate the environment 
according to any standard. For example., the environment estimating unit may estimate the environment based upon 

45 a used language, time zone, local information, phone number or the like set to an OS installed to this device or to the 
device itself. Further, the environment estimating unit may estimate the environment based upon the information at- 
tached to the captured image (e.g., EXIF or other header information). The selecting unit selects the approximate 
equation stored in the approximate equation storing unit so as to associate with the estimated environment. Further, 
in the thirteenth aspect, the pupil color estimating unit estimates a pupil color based upon the specified skin color and 

so using the selected approximate equation. 

[0050] That a pupil color is different depending upon the race is based upon an experiential fact. Further, that a race 
component is different depending upon the environment (such as a country or area) is also based upon an experiential 
fact. Therefore, there is a possibility that the race of the subject is different in the image captured in the different 
environment, so that estimating a pupil color by using the same approximate equation possibly degrades the precision . 

55 However the thirteenth aspect makes it possible to estimate a more suitable pupil color according to the environment 
in which this device operates. Therefore, more natural pupil color can be estimated. 
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[Others] 

[0051 ] The first to thirteenth aspects may be realized in such a manner that a program is executed by an information 
processing device. Specifically, the above-mentioned operations and effects may be obtained by a program for exe- 

s cuting each process executed by each units in the first to thirteenth aspects to an information processing device or by 
a recording medium having the program recorded thereon. Further, the above-mentioned operations and effects may 
be obLained by a method wherein an information processing device executes each process executed by each units in 
the first to thirteenth aspects. Further the first to thirteenth aspects may be provided at an imaging device such as a 
digital still camera or digital a video camera, a digital photo printer including a minilab machine, or other image process- 

10 jng device. 

[0052] The present invention can estimate a color of a pupil in which a color contrast is considered for a skin color 
in an inputted image. Therefore, an original pupil color can be estimated even though the original pupil color is lost 
from an image due to the occurrence of a red-eye phenomenon or a gold-eye phenomenon. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

[0053] 

Fig. 1 shows an example of a functional block of a pupil color estimating device according to a first embodiment. 
20 Fig. 2 shows a view of an example of a face rectangle. 

Fig. 3 shows a view of an example of a region specified upon obtaining a representative value of a skin color. 
Fig. 4 shows a flowchart of an operational example of the pupil color estimating device according to the first em- 
bodiment. 

Fig. 5 shows a view of an example of a functional block of a pupil color estimating device according to a second 
25 embodiment. 

Fig. 6 shows a flowchart of an operational example of the pupil color estimating device according to the second 
embodiment. 

Fig. 7 shows a view of a functional block of a pupil color estimating device according to a third embodiment. 
Fig. 8 shows a flowchart of an operational example of the pupil color estimating device according to the third 
embodiment. 

Fig. 9 shows a view of an example of a functional block of a pupil color estimating device according to a fourth 
embodiment. 

Fig. 10 shows an example of setting features. 
Fig. 11 shows an example of the Gabor filter. 
^5 Fig. 12 shows an example of a binary tree search. 

Fig. 13 shows a flowchart of an operational example of the pupil color estimating device according to the fourth 
embodiment. 

Fig. 14 shows a view of an example of a functional block of a pupil color estimating device according to a fifth 
embodiment. 

■*o Fig. 15 shows a flowchart of an operational example of the pupil color estimating device according to the fifth 

embodiment. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0054] Subsequently, a pupil color estimating device will be explained with reference to drawings. It should be noted 
that the following explanation about the pupil color estimating device is an illustration, so that its construction is not 
limited to the following explanation. 

[First Embodiment] 

50 

[System configuration] 

[0055] Firstly explained is a pupil color estimating device 1 a that is a first embodiment of the pupil color estimating 
device. The pupil color estimating device 1 a has a CPU (central processing unit) , main memory (RAM: Random Access 
55 Memory) , auxiliary memory or the like, each of which is connected via a bus. The auxiliary memory is configured by 
using a nonvolatile memory unit. The nonvolatile memory unit here indicates a so-called ROM (Read-Only Memory: 
including EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable 
Read-Oniy Memory) , mask ROM or the like) , FRAM (Ferroelectric RAM) , hard disk or the like. 
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[0056] Fig. 1 is a diagram showing a functional block of the pupil color estimating device 1a. Various programs (OS, 
application, or the like) stored in the auxiliary memory is loaded to the main memory to be executed by the CPU, 
whereby the pupil color estimating device 1 a functions as a device including a memory section 2, face detecting section 
3. skin color specifying section 4 and pupil color estimating section 5a. A program is executed by the CPU, so that the 

s face detecting section 3, skin color specifying section 4 and the pupil color estimating section 5a are realized. Further, 
each of the face delecting section 3, skin color specifying section 4 and the pupil color estimating section 5a may be 
configured as a dedicated chip. Moreover, some are suitable for a hardware and others are suitable for a software 
according to the content of a process executed by each processing section. Therefore, these may be mounted as a 
hybrid of a hardware and a software. Each functional section provided at the pupil color estimating device 1a will be 

10 explained hereinbelow. 

[Memory section] 

[0057] The memory section 2 stores an image inputted to the pupil color estimating device 1 a. The memory section 
15 2 may be configured by using a so-called RAM or a so-called ROM. 

[0058] The image inputted to the memory section 2 includes, for example, an image captured by an imaging device 
not shown, an image transmitted from an information processing device not shown via an interface, an image read 
from a recording medium not shown, or the like. 

20 [Face detecting sectionl 

[0059] The face detecting section 3 executes a face detecting process to an image, which is a subject to be processed 
by the pupil color estimating device 1a, among images stored in the memory section 2. In the face detecting process, 
the face detecting section 3 reads the image to be processed from the memory section 2, and a face rectangle 6 is 
25 detected from the read image. Fig. 2 is a view showing an example of a face rectangle (face rectangle 6). The face 
rectangle 6 is a rectangle including a face section of a subject person. 

[0060] When detecting the face rectangle 6, the face detecting section 3 outputs face rectangle information. The 
face rectangle information is information showing a size or position of the face rectangle 6. For example, the face 
rectangle information shows the width of the face rectangle 6 and the coordinates at the upper-left corner. The other 
30 processing sections specify the position of the face of the subject person in the image to be processed by this face 
rectangle information. 

[0061] The face detecting process by the face detecting section 3 may be configured, for example, to detect a face 
with a template matching using a reference template corresponding to an outline of a whole face. Further, it may be 
configured to detect a face with a template matching based upon components (eyes, nose, mouth or the like) of the 
35 face Moreover it may be configured to detect a face based upon the top of the hair on the head detected by a chroma- 
key process. Further the face detecting section 3 may detect a region close to the skin color, thereby detecting this 
region as a face. Moreover the face detecting section 3 may learn by a teacher signal by using a neutral network to 
detect a face-like region as a face. Further, the face detecting process by the face detecting section 3 may be realized 
by any other known methods. 

40 

[Skin color specifying section] 

[0062] The skin color specifying section 4 obtains a representative value of the skin color of the subject person from 
the region of the face rectangle 6 detected by the face detecting section 3. This process is called a skin color specifying 
•*5 process hereinbelow. The skin color specifying section 4 obtains, for example, statistics of a color value of the pixel 
included in the face rectangle 6 as a representative value in the skin color specifying process. More preferably, the 
skin color specifying section 4 is configured to obtain the statistics of a color value of a predetermined pixel among 
pixels included in the face rectangle 6. 

[0063] "The predetermined pixel" is, for example, a pixel included in the region specified from the face rectangle 
50 information of the face rectangle 6. More specifically, for example, it may be a pixel in the region geometrically specified 

from the face rectangle information of the face rectangle 6. Fig. 3 is a view showing an example of such a region. In 

Fig. 3. a region in a circle is specified as an example of such region, this circle having a diameter definedby 3/4 of the 

width of the face rectangle 6 and the center defined by the center of the face rectangle 6 ("p" in Fig. 3). 

[0064] Further "the predetermined pixel" may be, for example, remaining pixels obtained by eliminating pixels that 
55 are causes of noise upon a skin color measurement among pixels included in the face rectangle 6. Examples of pixels 

that are causes of the noise include a face component (e.g., eyebrow, bulb of an eye, lip, nostril) or a shiny section of 

a skin. 

[0065] Further "the predetermined pixel" may be, for example, a pixel included inside of the face outline specif iedby 
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a process such as an edge extraction, among pixels included in the face rectangle 6. Moreover, the specifying methods 
of the aforesaid "predetermined pixel" may be combined with each other according to need. 

[0066] Then, the skin color specifying section 4 executes a statistical process based upon the predetermined pixel 
to obtain the representative value of the skin color of the subject person. The statistical process may be a simple 

5 process such as an intermediate value, average, mode or the like. Moreover, a process for eliminating a cause of noise 
by dividing a range from a histogram with a method such as 3a may further be executed in the statistical process. 
[0067] The skin color specifying section 4 may be configured to obtain the representative value of the skin color of 
the subject person by a method different from the above-mentioned methods upon executing the skin color statistical 
process. For example, the skin color specifying section 4 may be configured to obtain a color value of a specified pixel 

fo in the face rectangle 6 as the representative value. 

[Pupil color estimating section] 

[0068] The pupil color estimating section 5a estimates the pupil color of the subject person based upon the repre- 
ss sentative value of the skin color obtained by the skin color specifying section 4. The pupil color estimated by the pupil 
color estimating section 5a is referred to as "pupil estimated color" hereinbelow. The pupil color estimating section 5a 
obtains the pupil estimated color based upon the transform function (approximate equation) of each color of RGB. This 
approximate equation indicates a correlation between the skin color in the image of the subject person and the pupil 
color in the image that is recognized to have no abnormality such as a red-eye occurring. This approximate equation 
20 is an equation pre-obtained as a pre-treatment. Firstly, the derivation of this approximate equation will be explained. 
[0069] Firstly collected as a sample image are a great number of images that are recognized to have no abnormality 
occurring on an image of a pupil such as a red-eye or gold-eye (i.e., "an image having no sense of incongruity", "an 
image recognized that the pupil of the subject person is normally imaged"). A statistical process is performed to these 
sample images, thereby obtaining a correlation between the skin color of the subject person in the image and the pupil 
25 color in the image recognized to have no abnormality. Specifically, a representative value Sn (Srn, Sgn, Sbn) of the 
skin color of the subject person in each sample image and a representative value En (Ern, Egn, Ebn) of the normal 
pupil color of this person are firstly obtained. These representative values are obtained by the same manner as, for 
example s the statistical process executed by the skin color specifying section 4. These representative values may be 
obtained by any other methods. 
so [0070] Subsequently, it is supposed that, for example, the transform function of each color can be approximated by 
a first-degree equation. Specifically, it is supposed that the transform function of each color is represented by the 
following equation. 

35 [Equation 1] 

Ec = Ac • Sc + Be 

[0071 ] in this equation, "c" represents any one of rgb. Further, "A" and "B" represent a coefficient. In the pre-treatment, 
jo the values of the coefficients "A" and "B" are determined by a least squares method based upon the representative 
value Sn of the skin color and the representative value En of the normal pupil color in each sample image. As described 
above, approximate equations with respect to each color of RGB, i.e., three approximate equations for estimating a 
normal pupil color from a skin color are obtained in the pre-treatment. 

[0072] Subsequently explained is a method for obtaining the pupil estimated color using the aforesaid approximate 
45 equation. The pupil color estimating section 5a substitutes the representative value of the skin color obtained by the 
skin color specifying section 4 with respect to the pre-obtained approximate equation, to thereby estimate a normal 
pupil color (i.e., pupil estimated color) of the subject. 

[0073] The aforesaid explanation is made about the case where the approximate equation of the first degree equation 
is obtained in the pre-treatment, but it is needless to say that the pre-treatment and the process at the pupil color 
50 estimating section 5a may be executed based upon the approximate equation other than the first degree equation. 

[Operational Example] 

[0074] Fig. 4 is a flowchart showing an operational example of the pupil color estimating device 1a. The operational 
55 example of the pupil color estimating device 1 a will be explained with reference to Fig. 4. 

[0075] Firstly, the face detecting section 3 detects the face of the subject person from the image to be processed 
(S01). Specifically, it detects the face rectangle 6 from the image to be processed to obtain the face rectangle infor- 
mation. Then, the skin color specifying section 4 specifies the representative value of the skin color based upon the 
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face rectangle information obtained by the face detecting section 3 (S02). Subsequently, the pupil color estimating 
section 5a estimates the pupil color of the person whose face is detectedby the face detecting section 3, by using the 
representative value of the skin color obtained by the skin color specifying section 4 and the pre-obtained approximate 
equation, thereby obtaining the pupil estimated color (S03). Then, the pupil color estimating section 5a outputs the 
5 estimated pupil estimated color (S04). 

[Operation/Effect] 

[0076] The pupil color estimating device 1 a estimates the normal color of the pupil of the subject person in the inputted 
10 image, i.e.. the pupil color that should be shown in the image ; from the skin color of the person based upon the pre- 
obtained approximate equation. Therefore, even though a problem such as a red-eye or gold-eye is caused on the 
inputted image, the normal pupil color of the subject person can be estimated independently of the abnormal pupil 
color s uch as the red-eye or gold-eye. Accordingly, a user can judge whether a correction should be made to the pupil 
section of the inputted image based upon the thus estimated pupil estimated color. Specifically, the user easily judges 
*5 whether the correction is made to the pupil section of the inputted image by comparing the pupil color of the inputted 
image and the pupil estimated color. Further the user can correct the pupil section of the inputted image based upon 
the thus estimated pupil estimated color, resulting in being capable of obtaining an image having no sense of incongruity 
and having a normal color of the pupil. 

[0077] The pupil color estimating device 1a described above may be applied to, for example, an imaging device such 
20 as a digital still camera or a video camera or to an information processing device provided with such an imaging device. 
The imaging device or the like can judge whether an abnormality such as a red-eye or gold-eye occurs on the captured 
image based upon the outputted pupil estimated color, since the pupil color estimating device 1a is applied thereto. 
Similarly, applying the pupil color estimating device 1a makes it possible to obtain a color to which the pupil image 
shouldbe corrected (i.e., pupil estimated color). Accordingly, these imaging devices or the like can make a suitable 
^5 correction, make an alarm to the user or present the pupil estimated color to the user, according to the judgement 
thereof 

[Second Embodiment] 
30 [System configuration] 

[0078] Subsequently explained is a pupil color estimating device 1 b that is the second embodiment of the pupil color 
estimating device. Fig. 5 is a view showing an example of a functional block of the pupil color estimating device 1b. 
The pupil color estimating device 1b is different from the pupil color estimating device 1a in that it is further provided 
55 with a pupil detecting section 7 and a judging section 8. The different points of the pupil color estimating device 1b 
from the pupil color estimating device 1a will be explained hereinbelow. It should be noted that the pupil detecting 
section 7 and the judging section 8 are realized in such a manner that a program is executed by a CPU. Further, each 
of the pupil detecting section 7and the judging section 8 may be configured as a dedicated chip. Moreover, each 
processing section may be mounted as a hybrid of a hardware and a software. 

40 

[Pupil detecting section] 

[0079] The pupil detecting section 7 executes a pupil detecting process for an image that is a subject to be processed 
by the pupil color estimating device 1 b, among images stored in the memory section 2. In the pupil detecting process, 
45 the pupil detecting section 7 reads from the memory section 2 the image in the face rectangle 6 detected by the face 
detecting section 3 among images that are subjects to be processed, and detects the pupil region from the read image. 
The pupil region means an image including the pupil of the subject person whose face is detected by the face detecting 
section 3. 

[0080] The pupil detecting process by the pupil detecting section 7 may be configured to detect a pupil with a template 
50 matching using a reference template corresponding to a shape of the pupil. In this case, the pupil detecting section 7 
may be configured to select a template corresponding to the diameter of the pupil estimated from the size of the face 
rectangle 6 detected by the face detecting section 3, to thereby execute the pupil detecting process. Further, in this 
case, the pupil detecting section 7 maybe configured to limit the region where the template matching is executed by 
estimating a rough pupil position from the position of the detected face rectangle 6. Further, the pupil detecting process 
55 by the pupil detecting section 7 may be realized by any methods other than the template matching. For example, the 
pupil detecting section 7 may be configured to detect the pupil by tracing the outline of the pupil after extracting an 
edge with respect to the inputted image. When detecting the pupil by the pupil detecting process, the pupil detecting 
section 7'outputs information showing the pupil region (hereinafter referred to as "pupil region information"). 
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[Judging section] 

[0081 J The judging section 8 specifies the color of each pixel showing the pupil in the inputted image based upon 
the outputted pupil region information by the pupil detecting section 7. Specifically, in the case where an abnormality 
5 such as a red-eye or gold-eye occurs on the inputted image, for example, the judging section 8 specifies the color in 
the image having the abnormality occurring. 

[0082] Subsequently, the judging section 8 judges the pupil region based upon the pupil region information, thereby 
judging, for each pixel included in the pupil region, whether the color possessed by the pixel is a normal color or not 
for the pupil. Specifically, the judging section 8 makes the aforesaid judgement by comparing the color specified in the 
10 inputted image and the pupil estimated color estimated by the pupil color estimating section 5a. 

[0083] The judging section 8 obtains a feature amount D for the pupil estimated color and the color of each pixel in 
the pupil region, for example, based upon the following equation. In this case, the judging section 8 obtains the feature 
amount D showing a degree of red for the pupil estimated color and the color in each pixel, thereby judging whether 
the red-eye phenomenon is present or not in particular 

15 



[Equation 2] 



r + g + b 



25 [0084] The judging section 8 judges the degree of red for each pixel by comparing the feature amount D for the pupil 
estimated color and the feature amount D for the color in each pixel in the pupil region. Then, the judging section 8 
judges that the abnormality {in this case, the red-eye phenomenon) occurs in this inputted image, in the case where 
pixels having the feature amount D greater than the feature amount D in the pupil estimated color are present in the 
pupil region at a higher ratio than the predetermined ratio. 

so [0085] It should be noted that the feature amount D used for the judgement by the judging section 8 may be obtained 
based upon the other definition. For example, the judging section 8 may be configured to obtain a feature amount D* 
for judging the degree of the gold-eye for making a comparison and judgement. 

[Operational Example] 

3.5 

[0086] Fig. 6 is a flowchart showing an operational example of the pupil color estimating device 1 b. The operational 
example of the pupil color estimating device 1 b will be explained with reference to Fig. 6. It should be noted that, among 
operations of the pupil color estimating device 1 b, the process same as that of the pupil color estimating device 1 a is 
given same numeral as that in the flowchart of Fig. 4 to omit its explanation. 

-to [0087] In the pupil color estimating device 1 b, after the pupil color estimating section 5a estimates the pupil estimated 
color (S03), the pupil detecting section 7 detects the pupil region (S05) . Then, the judging section 8 judges whether 
an abnormality such as a red-eye or gold-eye occurs on the pupil in the inputted image based upon the pupil estimated 
color estimated by the pupil color estimating section 5a and the color of each pixel in the pupil region detected by the 
pupil detecting section 7 (S06). Then, the judging section 8 outputs the judging result (S07). It should be noted that 

^5 the process for detecting the pupil region by the pupil detecting section 7 (SOS) may be executed before the process 
for estimating the pupil estimated color by the pupil color estimating section 5a (S03). 

[Operation/Effectl 

50 [0088] The pupil color estimating device 1 b judges whether an abnormality such as a red-eye phenomenon or gold- 
eye phenomenon occurs on the image of the pupil in the inputted image based upon the pupil estimated color estimated 
based upon the same process as the pupil color estimating device 1 a. Therefore, a user can obtain an objective judge- 
ment as to whetherthe abnormality such as the red-eye phenomenon or gold-eye phenomenon occurs on the inputted 
image. Accordingly, the user can easily judge whether a correction should be made or not to the inputted image, in the 

55 case where a correction is made to the image that is recognized to have an abnormality, for example. Similarly, in the 
case where the user desires to re-obtain the inputted image when the abnormality occurs, he/she can easily judge 
whether the inputted image should be re-obtained or not (for example, whether a process such as image-capture or 
download should be performed again). 
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[0089] The pupil color estimating device 1 b described above may be applied to, for example, an imaging device such 
as a digital still camera or a video camera or to an information processing device provided with such imaging device, 
like the pupil color estimating device 1a. Whether an abnormality such as a red-eye or gold-eye occurs on the captured 
image can be judged by applying the pupil color estimating device 1 b, and according to this judgement, a suitable 
correction can be made or an alarm can be given to the user. 

[Modified Example] 

[0090] The judging section 8 may be configured to make a judgement independently for each of the right and left 

io pupils of the subj ect person and output the judging result for each pupil. 

[0091] Further, in the case where faces of plural subj ect persons are detected in the inputted image, the pupil color 
estimating device 1b may be configured to execute the judging process for all persons, or execute the judging result 
for only the subject person satisfying the predetermined condition (for example, a person having the face rectangle 6 
greater than the predetermined size, a person positioned at the vicinity of the center of the screen, a specific person 

15 judged by a face authentication). 

[Third Embodiment] 
[System configuration] 

20 

[0092] Subsequently explained is a pupil color estimating device 1c that is the third embodiment of the pupil color 
estimating device. Fig. 7 is a view showing an example of a functional block of the pupil color estimating device 1c. 
The pupil color estimating device 1c is different from the pupil color estimating device 1b in that it is further provided 
with a correcting section 9. The different points of the pupil color estimating device 1c from the pupil color estimating 
25 device 1 b will be explained hereinbelow. It should be noted that the correcting section 9 is realized in such a manner 
that a program is executed by a CPU . Further, the correcting section 9 may be configured as a dedicated chip. Moreover, 
each processing section may be mounted as a hybrid of a hardware and a software. 

[Correcting section] 

30 

[0093] The correcting section 9 executes a correcting process based upon the judging result by the judging section 
8. Specifically, the correcting section 9 executes the correcting process in the case where an abnormality is judged to 
occur on the pupil image in the inputted image by the judging section 8. The correcting section 9 corrects the color of 
the pixel in the pupil region detected by the pupil detecting section 7 based upon the pupil estimated color estimated 
35 by the pupil color estimating section 5a. One example of the correcting process executed by the correcting section 9 
will be explained hereinbelow. 

[0094] The correcting section 9 replaces the color of the pixel that is the subject to be corrected with the pupil esti- 
mated color while keeping its brightness. Here, the correcting section 9 obtains the brightness Y of the pixel to be 
corrected based upon the following equation. 

40 

[Equation 3] 
Y = 0.299R + 0.587G + 0.1 1 4B 

45 

[0095] It should be noted that each value of R, G and B is an integer from 0 to 255. The correcting section 9 obtains 
the RGB value (hereinafter referred to as Nr, Ng, Nb) from which the brightness same as the brightness Y of the pixel 
to be corrected can be obtained, while keeping the ratio of each value of R, G and B in the pupil estimated color (i.e., 
the values of Er, , Eg, Eb obtained from the approximate equation) . The correcting section 9 executes the correcting 
50 process by giving thus obtained values of Nr, Ng and Nb to the pixel to be corrected. 

[Operational Example] 

[0096] Fig. 8 is a flowchart showing an operation example of the pupil color estimating device 1c. The operational 
55 example of the pupil color estimating device 1c will be explained with reference to Fig. 8. It should be noted that, among 
operations of the pupil color estimating device 1c, the process same as that of the pupil color estimating device 1b is 
given same numeral as that in the flowchart of Fig. 6 to omit its explanation. 

[0097] In the pupil color estimating device 1c, after the judging section 8 judges about the occurrence of the abnor- 
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malfty (S06) , the correcting section 9 judges whether the correcting process is necessary or not based upon the judging 
result. In the case where the correcting process is necessary (SOB-YES), i.e., in the case where the judging section 8 
judges that the abnormality occurs on the pupil image, the correcting section 9 executes the correcting process (S09), 
and then, the process is ended. On the other hand ; in the case where the correcting process is unnecessary (S08-NO), 
? i.e.. in the case where the judging section 8 judges that the abnormality does not occur on the pupil image, the process 
is ended. 

[Operation/Effect] 

10 [0098] In the pupil color estimating device 1c, whether the correcting process is required or not to the pupil section 
of the inputted image is judged based upon the judging result of the judging process by the judging section 8 executed 
based upon the same process as the pupil color estimating device 1b. Then, according to this judging result, the 
correcting process using the pupil estimated color estimated based upon the same process as the pupil color estimating 
device 1 a is executed. Therefore, the pupil color of the subject person is corrected to a normal color without troubling 

15 the user with respect to the image in which the abnormality such as a red-eye phenomenon or gold-eye phenomenon 
occurs 

[0099] Further, in the pupil color estimating device 1c, a color based upon the pupil estimated color is determined, 
while maintaining the brightness of the pixel to be corrected in the inputted image, upon executing the correcting 
process. Specifically, the color replacement is executed with the ratio of RGB of the pupil estimated color or the original 
20 pixel brightness maintained. Therefore, more natural correcting process can be executed, different from the case of 
simply replacing with the pupil estimated color. Specifically, it is possible to naturally express a spherical shape of a 
bulb or a sense of wettability on the surface of the bulb in the image of the pupil after the correction without losing an 
effect due to a natural phenomenon caused on the surface of the pupil such as a so-called catch light. Accordingly, 
high-quality image correction can be realized. 

25 

[Modified Example] 

[0100] The pupil color estimating device 1c may be configured to omit the judging section 8. In this case, the correcting 
section 9 is configured to execute the correcting process independently of the judging result by the judging section 8. 
30 in this case, it may be configured, for example, that the correcting process is executed based upon the judgement by 

a user. 

[0101] By this configuration, the correcting process is executed even to a pupil having the gold-eye phenomenon or 
a pupil giving a sense of incongruity due to the reason of too black, even though the judging section 8 is configured to 
judge only about the red-eye phenomenon, for example. Accordingly, it is possible to cope with the abnormality thai 

25 cannot be judged by the judging section 8. 

[0102] Further, the judging section 8 may be configured to make a judgement independently for each of the right and 
left pupils of the subject person and execute the correcting process to only the pupil that the correction is necessary. 
This configuration prevents the correcting process from being executed to the pupil that should not originally be cor- 
rected, and enables a high-speed process. 

40 [0103] Further, in the case where faces of plural subject persons are detected in the inputted image, the pupil color 
estimating device 1c may be configured to execute the judging process for all persons, or execute the judging result 
for only the subject person satisfying the predetermined condition (for example, a person having the face rectangle 6 
greater than the predetermined size, a person positioned at the vicinity of the center of the screen, a specific person 
judged by a face authentication). 

45 

[Fourth Embodiment] 
[System configuration] 

50 [0104] Subsequently explained is a pupil color estimating device 1d that is the fourth embodiment of the pupil color 
estimating device. Fig. 9 is a view showing an example of a functional block of the pupil color estimating device 1d. 
The pupil color estimating device 1 d is different from the pupil color estimating device 1a in that it is further provided 
with a race estimating section 10. Further, the pupil color estimating device 1d is also different from the pupil color 
estimating device 1a in that it is provided with a pupil color estimating section 5d instead of the pupil color estimating 

55 section 5a. It should be noted that the pupil color estimating section 5d and the race estimating section 1 0 are realized 
in such a manner that a program is executed by a CPU. Further, the pupil color estimating section 5d and the race 
estimating section 1 0 may be configured as a dedicated chip. Moreover, each processing section may be mounted as 
a hybrid of a hardware and a software. 
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[Race estimating section] 

[0105] The race estimating section 1 0 estimates which race the person's face detected by the face detecting section 
3 belongs to by using an image. The race estimating section 10 may be realized by any technique, so long as it is a 
5 technique for estimating the race of the subject from the image. Explained hereinbelow Is a specific example of a race 
estimating technique applicable to the race estimating section 10. 

[01 06] The race estimating section 1 0 obtains a feature amount with respect to the face detected by the face detecting 
section 3 : and performs a pattern recognition based upon the feature amount to thereby estimate the attribute. In order 
to realize the process described above ; the race estimating section 10 includes a feature extracting section 11 and 
10 attribute estimating section 12. Each process executed by each section will be explained hereinbelow. 

[[Feature extracting section]] 

[01 07] At first, the feature extracting section 1 1 sets plural features in the face detected by the face detecting section 
15 3 (feature setting process). Then, the feature extracting section 11 obtains a feature amount of each feature as the 
feature amount of the face of this subject based upon the features set in the feature setting process (feature amount 
obtaining process) . The feature setting process and the feature amount obtaining process will be explained herein- 
below. 

20 {[[Feature setting processll) 

[0108] In the feature setting process, the feature extracting section 11 firstly detects organs of the detected face. 
The organs of the face mean, for example, eyes, nose, nostrils, mouth (lip), eyebrows, chin, forehead or the like. The 
feature extracting section 1 1 may detect any organs in the face and may detect plural organs. For example, the feature 

25 extracting section 11 is configured to detect both eyes and mouth in the face of the subject. 

[0109] Then, the feature extracting section 11 converts the detected face image into a gray scale image. Further, 
the feature extracting section 11 executes an angular normalization or size normalization of the detected face image 
based upon the positional relationship of the organs of the detected face. These processes are collectively referred to 
as a pre-process. Further, the process for converting the image into a gray scale may be executed during the process 

30 at the face detecting section 3 or at any time during the feature setting process. 

[01 1 0] Subsequently, the feature extracting section 1 1 sets positions of the plural features based upon the positions 
of the organs (hereinafter referred to as "fixation point", e.g., the point showing both eyes and mouth) in the detected 
face. The feature extracting section 11 sets features densely at the section close to the fixation point while roughly at 
the section away from the fixation point. 

35 [0111] Fig. 1 0A is a view showing the face of the subject detected by the face detecting section 3. Fig. 1 0B is a view 
showing examples of plural features set by the feature setting process. In Fig. 10B, black-painted circles indicate 
fixation points, while slanted circles indicate features set based upon the fixation points. The fixation points may be 
handled as features in the feature amount obtaining process explained below. 

[0112] The feature setting process described above can be realized by, for example, applying the Retina sampling 
40 disclosed in the following thesis. 

[0113] F. Smeraldiand J.Bigun, "Facial features detection by saccadic exploration of the Gabor decomposition", In- 
ternational Conference on Image Processing, IPC-98, Chicago, October 4-7, volume 3, pages 163-167, 1998. 

{[[Feature amount obtaining process]]} 

45 

[0114] In the feature amount obtaining process, the feature extracting section 11 folds the Gabor filterto each feature 
set by the feature setting process. Specifically, the feature extracting section 11 executes the GaborWavelets Trans- 
formation (GWT) to each feature. Fig. 1 1 is an example (real part) of the Gabor filter used in thefeature amount obtaining 
process. The feature extracting section 11 obtains as the feature amount the periodicity and directivity of the concen- 
50 tration feature in the vicinity of the feature by folding plural Gabor filters having the resolution and orientation changed 
as shown in Fig. 11 . 

[0115] The equation 4 represents the Gabor filter. In the use of the Gabor filter, changing the value of k and 6 in the 
equation makes it possible to obtain optional periodicity and directivity as a feature amount from the concentration 

feature. 

55 
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[0116] The feature extracting section 11 gives the feature amount of each feature obtained by the feature amount 
obtaining process to the attribute estimating section 12. It should be noted that the feature extracting section 11 may 
be configured to perform the process with respect to all faces satisfying a predetermined condition among faces de- 
tected by the face detecting section 3. The predetermined condition means a condition such as a face having a size 
greater than a predetermined size, a predetermined position (e.g., a region at the center of the image), a face with a 
predetermined direction (e.g., facing frontward) or the like. 

[[Attribute estimating section]] 

[01 17] The attribute estimating section 1 2 estimates attribute information of the subject detected by the face detecting 
section 3 based upon the feature amount of each feature given from the feature extracting section 11 . The attribute 
information is information relating to this person including, for example, a race, age, sex or the like as an item. The 
attribute estimating section 1 2 estimates the attribute information of the subject by inputting the feature amount of each 
feature to a discriminator of a pattern recognition in which the learning process is completed in advance. The attribute 
estimating section 12 executes the process by using Support Vector Machine (SVM) as a discriminator of a pattern 
recognition. The Support Vector Machine will be explained hereinbelow by taking the case of estimating the race in 
the attribute information as an example. 

[0118] The Support Vector Machine is a technique wherein a hyperplane passing through the center of the learning 
dala belonging lo two classes is defined as a border of the discrimination, thereby performing a pattern recognition. 
The discriminator in the Support Vector Machine uses the discrimination function represented by the equation 5, thereby 
estimating which class the inputted data (the feature amount at all features here) belongs to. 



[Equation 5] 



45 



f(x) = ya i y i K(x,x i )+b 



[0119] In the equation 5, 1 represents the number of the learning data sorted by the learning process, i.e.. the number 
of the learning data used for the attribute estimating process by the attribute estimating section 12. a t represents 
50 Lagrange multiplier. x f and yj represent the learning data. y { has either one of "-1" or "1", showing which class Xj belongs 
to. b represents a bias term. i.e. , parameter. These values are determined by the learning process, whereby the attribute 
estimating section 1 2 stores the result of the learning process. 

[0120] Moreover, in the equation 5, K represents a kernel function. There has been proposed a non-linear expansion 
of the Support Vector Machine that high-dimensionally maps the inputted data in a non-linear manner by using the 
55 kernel function. This makes it possible to construct a discriminator more effective for the actual problem. The repre- 
sentative kernei functions described above include a polynomial kernel (see equation 6) or Gaussian kernel (see equa- 
tion 7) or the like. Any kernel function may be applied in the attribute estimating section 12. 
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[Equation 6] 

5 



10 



20 



35 



50 



[Equation 7] 



i^(jc,Jc ( .)-exF 



[01 21 ] The Support Vector Machine is a learning method that composes a discriminator discriminating two classes. 
In order to discriminate (estimate) plural races (more than three races) , plural Support Vector Machines are required 
to be combined, The attribute estimating section 12 applies a binary tree search, thereby realizing a multi-clustering 
25 using the Support Vector Machine. Fig. 1 2 is a view showing an example of a binary tree search applied for the race 
estimating process in the attribute estimating section 12. The explanation is made here by taking, as an example, a 
process for estimating which race of the subject is, Caucasoid, Negroid or Mongoloid. It may be configured to include 
other races as a candidate according to the design. 

[01 22] The attribute estimating section 1 2 firstly estimates whether the subject person is the Negroid or not by using 
30 the Support Vector Machine. In the case where the subject person is estimated not to be the Negroid, the attribute 
estimating section 12 then estimates whether the subject person is the Mongoloid or not (or the Caucasoid or not). 
The attribute estimating section 1 2 performs the estimating process about each term of the attribute information other 
than the race, i.e., age or sex, by similarly executing the binary tree search. Then, the attribute estimating section 12 
gives the estimated attribute information to the pupil color estimating section 5d. 



[Pupil color estimating section] 



[0123] The pupil color estimating section 5d estimates the pupil color of this subject person based upon the repre- 
sentative value of the skin color obtained by the skin color specifying section 4 and the attribute information obtained 
jo by the race estimating section 10. The process executed by the pupil color estimating section 5d will be specifically 
explained hereinbelow. 

[0124] The approximate equation (corresponding to the equation 1 ) associated with each race is obtained in advance. 
Specifically, a sample image including the subject of each race is prepared as each sample image, whereby the ap- 
proximate equation corresponding to each race is obtained. Then, the approximate equation corresponding to each 
45 race is stored by the pupil color estimating section 5d. When the pupil color of the subject is estimated, the pupil color 
estimating section 5d selects one approximate equation according to the result of the estimation by the race estimating 
section 1 0. Then, the pupil color estimating section 5d estimates the pupil color according to the output from the skin 
color specifying section 4 by using the selected approximate equation. The process in this case is the same as the 
process of the pupil color estimating section 5a. 



[Operational Example] 



[01 25] Fig. 1 3 is a flowchart showing an operational example of the pupil color estimating device 1 d. The operational 
example of the pupil color estimating device 1d will be explained with reference to Fig. 13. It should be noted that, 
55 among operations of the pupil color estimating device 1 d, the process same as that of the pupil color estimating device 
1a is given same numeral as that in the flowchart of Fig. 4 to omit its explanation. 

[01 26] In the pupil color estimating device 1 d, the race estimating section 1 0 estimates the race of the subject (S1 0) 
after the skin color specifying section 4 specifies the skin color (S02). Subsequently, the pupil color estimating section 
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5d selects the approximate equation used for the process based upon the specified skin color and the estimated race 
(S11 ). The pupil color estimating section 5d estimates the original pupil color of the subject according to the race by 
using the selected approximate equation (S12). Then, the pupil color estimating section 5d outputs the pupil estimated 
color that is the result of the estimation, and ends the process (S04). 

5 

[Operation/Effect] 

[0127] The approximate equation used in the pupil color estimating device 1a has depended upon the content of the 
set of the sample images used upon pre-obtaining this approximate equation. Accordingly, in the case where the set 

io of the sample images include, for example, many Japanese images, the approximate equation is obtained as is suitable 
for the Japanese. Therefore, in the case where such approximate equation is applied as it is to the image of the 
Europeans and the Americans, there arises a problem that the obtained pupil estimated color has less blue color and 
too much black color, thusunsuitable. To cope with this problem, in the pupil color estimating device 1d, the race esti- 
mating section 10 estimates the race of the subject and the pupil color estimating section 5d selects the approximate 

15 equation used according to the race that is the result of the estimation. Accordingly, a suitable pupil color according to 
the race of the subject can be obtained as the result of the estimation, when the pupil color of the subject is estimated 
by the pupil color estimating section 5d. 

[Modified Example] 

20 

[01 28] The pupil color estimating device 1 d may be configured to further have the pupil detecting section 7 and the 
judging section 8 like the pupil color estimating device 1b. Further, the pupil color estimating device 1d may be con- 
figured to be further provided with the correcting section 9 like the pupil color estimating device 1c. 
[0129] Moreover, the selection of the approximate equation used for the estimation of the pupil color is executed by 
25 the pupil color estimating section 5d in the pupil color estimating device 1 d, this selection may be executed by the race 
estimaiing section 10. 

[Fifth Embodiment] 

3o [System configuration] 

[0130] Subsequently explained is a pupil color estimating device 1e that is the fifth embodiment of the pupil color 
estimating device. Fig. 14 is a view showing an example of a functional block of the pupil color estimating device 1e. 
The pupil color estimating device 1e is different from the pupil color estimating device 1a in that il is further provided 
25 with an environment estimating section 13. Further, the pupil color estimating device 1e is also different from the pupil 
color estimating device 1 a in that it is provided with a pupil color estimating section 5e instead of the pupil color esti- 
mating section 5a. It should be noted that the pupil color estimating section 5e and the environment estimating section 
13 are realized in such a manner that a program is executed by a CPU. Further, the pupil color estimating section 5e 
and the environment estimating section 1 3 may be configured as a dedicated chip. Moreover, each processing section 
may be mounted as a hybrid of a hardware and a software. 

[Environment estimating section] 

[0131] The environment estimating section 13 estimates the environment in which the pupil color estimating device 
4 5 1 e is installed or the environment in which it operates (environment estimating process). Then, the environment esti- 
mating section 13 obtains an approximate equation used in the pupil color estimating section 5e based upon the result 
of the estimation (approximate equation obtaining process). The environment estimating process and the approximate 
equation obtaining process will be explained hereinbelow. 

50 [[Environment estimating process]] 

[0132] In the environment estimating process, the environment estimating section 13 estimates the environment in 
which the pupil color estimating device 1 e is installed or the environment in which it operates according to a predeter- 
mined reference. For example, the environment estimating section 13 may estimate the environment with the desig- 
55 nation of the phone number of local information or maintenance information set in an OS (Operating System) or the 
like defined as a reference. Further, the environment estimating section 1 3 may estimate the environment with the time 
zone set in the OS or in the device defined as a reference. Moreover, the environment estimating section 13 may 
estimate the environment with the used language set in the OS or in the device defined as a reference. 
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[[Approximate equation obtaining process]] 

[0133] In the approximate equation obtaining process, the environment estimating section 13 obtains the approxi- 
mate equation used in the pupil color estimating section 5e based upon the result of the estimation in the environment 
5 estimating process. The environment estimating section 13 stores in advance plural approximate equations. These 
plural approximate equations are stored so as to associate with each environment. Specifically, one approximate equa- 
tion is obtained in the approximate equation obtaining process with respect to the environment estimated by the envi- 
ronment estimating process. 

[01 34] Each approximate equation stored in the environment estimating section 1 3 is the same approximate equation 
10 as the equation 1 ; and is calculated in advance so as to associate with each environment. For example, the approximate 
equation corresponding to the environment having a race composing ratio such as 70 % of Mongoloid, 20 % of Cau- 
casoid and 1 0 % of Negroid is obtained as follows. At first, sample images corresponding to the race composing ratio 
are prepared. In the case where 1 000 sample images are used in total, prepared images include 700 Mongoloid images, 
200 Caucasoid images and 100 Negroid images. Any of the prepared images has normal pupil image. The value of 
is the skin color and the value of the pupil color are plotted every color of RGB in a graph with respect to all sample 
images, whereby the approximate equation corresponding to the equation 1 with respect to each color of RGB is 
obtained by a least squares method. In other words, a sample set is formed based upon the race composing ratio 
according to each environment, thereby obtaining the approximate equation associated with each environment. 

20 [Pupil color estimating section] 

[0135] The pupil color estimating section 5e estimates the pupil color by using the approximate equation obtained 
by the environment estimating section 13. The specific process in this case is the same as the process of the pupil 
color estimating section 5a, except for the use of the approximate equation obtained by the environment estimating 

25 section 13. 

[Operational Example] 

[0136] Fig. 15 is a flowchart showing an operational example of the pupil color estimating device 1e. The operational 
30 example of the pupil color estimating device 1e will be explained with reference to Fig. 15. It should be noted that, 
among operations of the pupil color estimating device 1 e, the process same as that of the pupil color estimating device 
1 a is given same numeral as that in the flowchart of Fig. 4 to omit its explanation. 

[0137] In the pupil color estimating device 1e, the environment estimating section 13 executes the environment 
estimating process to estimate the environment (S13) after the skin color specifying section 4 specifies the skin color 
35 (S02) Then, the environment estimating section 1 3 further executes the approximate equation obtaining process to 
thereby obtain the approximate equation (S14). Thereafter, the pupil color estimating section 5e estimates the pupil 
estimated color by using the approximate equation obtained by the approximate equation obtaining process (S15), 
and outputs this color (S04). 

40 [Operation/Effect] 

[0138] The pupil color estimating device 5e estimates the environment in which the pupil color estimating device 1e 
is installed or the environment in which it operates, without executing the estimating process about the race of the 
subject, to thereby obtain an approximate equation suitable for the environment. Therefore, the effect same as that 

^5 obtained by the pupil color estimating device 1 d can be obtained, although the specific process is different. Specifically, 
in the case where the pupil color estimating device 1e is operated in Japan, for example, the approximate equation 
calculated based upon the race composing ratio of the Japanese is used to estimate a pupil color. On the other hand, 
in the case where the pupil color estimating device 1 e is operated in a country having many Caucasoid, the approximate 
equation calculated based upon its race composing ratio is used to thereby estimate a pupil color. Therefore, a suitable 

50 pupil estimated color according to the race of the subject can be obtained. Different from the pupil color estimating 
device Ida complicated process such as the estimation of the race of the subject is unnecessary in this case, thereby 
being capable of achieving a high-speed process, reduction in cost, miniaturization of the device or the like in the pupil 
color estimating device 1e. 

55 [Modified Example] 

[0139] The pupil color estimating device 1e may be configured to further have the pupil detecting section 7 and the 
judging section 8 like the pupil color estimating device 1b. Further, the pupil color estimating device 1e may be con- 
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figured to be further provided with the correcting section 9 like the pupil color estimating device 1c. 

[01 40] Moreover, the approximate equation obtaining process may be executed by the pupil color estimating section 

5e in the pupil color estimating device 1 e. 

[0141] Further the pupil color estimating devices 1a to 1e may be installed to a digital camera or a digital photo 
printer including a minilab machine to be used for performing an image correction of a pupil color. 
[0142] Moreover the environment estimating section 13 may estimate the environment in which the inputted image 
is capLured based upon the information (e.g., EXIF or other header information) attached to the inputted image. 



Claims 

1. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) comprising: 

skin color specifyingmeans (4) for specifying a skin color of a subject person in an inputted image; and 
pupil color estimating means (5a, 5d, 5e) for estimating a color of a pupil of the person that should be shown 
in said image based upon the skin color specified by said skin color specifying means (4). 

2. A pupil color estimating device (1a, 1b, 1c. 1 d. 1 e) comprising: 

face detecting means (3) for detecting a face region including a part of a face or a whole face of a subject 
person in an inputted image; 

skin color specifying means (4) for specifying a skin color of the person from a color of a pixel included in the 
face region detected by said face detecting means (3); and 

pupil color estimating means (5a, 5d, 5e) for estimating a color of a pupil of the person that should be shown 
in said image based upon the skin color specified by said skin color specifying means (4). 

3. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) claimed in Claim 2, further comprising: 

pupil detecting means (7) for detecting a pupil region including a part of the pupil or whole pupil of the person 
detected by said face detecting means (3); and 

judging means (8) for judging whether an abnormality occurs on the color of the pupil of the person in said 
image by comparing the color of the pupil estimated by said pupil color estimating means (5a, 5d, 5e) and a 
color specified from the pupil region detected by said pupil detecting means (7). 

4. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) claimed in Claim 3, wherein said judging means (8) judges 
whether the pupil section of the person in said image becomes a red-eye or not by comparing the the pupil color 
estimated by said pupil color estimating means (5a, 5d, 5e ) and the color specified from the pupil region detected 
by said pupil detecting means (7). 

5. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) claimed in Claim 2, further comprising: 

pupil detecting means (7) for detecting a pupil region including a part of the pupil or whole pupil of the person 
detected by said face detecting means (3); and 

correcting means (9) for correcting the pupil color based upon the pupil color estimated by said pupil color 
estimating means (5a ; 5d, 5e), with respect to the pupil region detected by said pupil detecting means (7). 

6. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) claimed in Claim 3, further comprising: 

correcting means (9) that corrects the pupil color based upon the pupil color estimated by said pupil color 
estimating means (5a, 5d, 5e), with respect to the pupil region detected by said pupil detecting means (7), in 
the case where it is judged by said judging means (8) that an abnormality occurs on the pupil color. 

7. A pupil color estimating device (1a, 1b, 1c. 1d, 1e) claimed in Claim 5, wherein said correcting means (9) makes 
a correction to the pixel in the pupil region detected by said pupil detecting means (7) with the brightness of the 
pixel maintained. 

8. A pupil color estimating device (1a, 1b, 1c, 1d, 1e) claimed in Claim 2, wherein said pupil color estimating means 
(5a. 5d ; 5e) estimates a pupil color based upon a correlation between a skin color and a pupil color, this correlation 
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obtained based upon the result of a statistical process of a skin color and a pupil color obtained from an image 
including a subject person that is recognized to have no abnormality occurring on the pupil section. 

9. A program that causes an information processing device to execute: 

5 

a step of specifying a skin color of a subject person in an inputted image; and 

a step of estimating a color of a pupil of the person based upon the skin color specified by said specifying step. 

10. A program that causes an information processing device to execute: 

10 

a step of detecting a face region of a subject person in an inputted image; 

a step of specifying a skin color of the person from a color of a pixel included in the face region detected in 
said detecting step; and 

a step of estimating a color of the pupil of the person based upon the skin color specified by said specifying step. 

15 

11 . A pupil color estimating method comprising: 

a step in which an information processing device specifies a skin color of a subject person in an inputted 
image: and 

20 a step in which an information processing device estimates a color of a pupil of the person based upon the 

skin color specified by said specifying step. 

12. A pupil color estimating method comprising: 

25 a step in which an information processing device detects a face region of a subject person in an inputted image; 

a step in which an information processing device specifies a skin color of the person from a color of a pixel 
included in the face region detected in said detecting step; and 

a step in which an information processing device estimates a color of the pupil of the person based upon the 
skin color specified by said specifying step. 

30 

13. A pupil color estimating device (1a, 1b, 1c. 1d, 1e) claimed in Claim 2, further comprising: 

approximate equation storing means that stores an approximate equation, used for estimating a pupil color, 
so as to associate with a race; 
35 race estimating means (10d) that estimates the race of said subject person; and 

selecting means that selects said approximate equation stored in said approximate equation storing means 
so as to associate with the estimated race, wherein 

said pupil color estimating means (5a, 5d, 5e) estimates the pupil color based upon said specified skin color 
and using said approximate equation. 

40 

14. A pupil color estimating device (1a 1b, 1c, 1d, 1e) claimed in Claim 2, further comprising: 

approximate equation storing means that stores an approximate equation, used for estimating a pupil color, 
so as to associate with an environment; 
45 environment estimating means (13) that estimates the environment in which this device operates or the envi- 

ronment in which the inputted image is captured, and 

selecting means that selects said approximate equation stored in said approximate equation storing means 
so as to associate with the estimated environment, wherein 

said pupil color estimating means (5a, 5d, 5e) estimates the pupil color based upon said specified skin color 
so and using said approximate equation. 

15. An imaging device comprising: 

imaging means that captures an image; 
55 face detecting means (3) for detecting a face region including a part of a face or a whole face of a subject 

person in an inputted image; 

skin color specifying means (4) for specifying a skin color of the person from a color of a pixel included in the 
face region detected by said face detecting means (3); 
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pupil color estimating means (5a, 5d, 5e) for estimating a color of a pupil of the person that should be shown 
in said image based upon the skin color specified by said skin color specifying means (4); 
pupil detecting means (7) for detecting a pupil region including a part of the pupil or whole pupil of the person 
detected by said face detecting means (3); and 

correcting means (9) for correcting the pupil color based upon the pupil color estimated by said pupil color 
eslimating means (5a, 5d ; 5e), with respect to the pupil region detected by said pupil detecting means (7). 

16. A digital photo printer comprising: 

image input means for inputting an image; 

face detecting means (3) for detecting a face region including a part of a face or a whole face of a subject 
person in an inputted image; 

skin color specifying means (4) for specifying a skin color of the person from a color of a pixel included in the 
face region detected by said face detecting means (3); 

pupil color estimating means (5a, 5d, 5e) for estimating a color of a pupil of the person that should be shown 
in said image based upon the skin color specified by said skin color specifying means (4); 
pupil detecting means (7) for detecting a pupil region including a part of the pupil or whole pupil of the person 
detected by said face detecting means (3); and 

correcting means (9) for correcting the pupil color based upon the pupil color estimated by said pupil color 
estimating means (5a, 5d, 5e). with respect to the pupil region detected by said pupil detecting means (7). 
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Fig. 2 
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Fig. 4 
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Fig. 6 
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Fig. 11 
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Fig. 13 
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Fig. 15 
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